Aiming at the shortcomings of the traditional radar signal detection algorithm, such as inaccuracy, large computation and slow speed, a signal detection algorithm of GST Hough transform based on time frequency clustering criterion is proposed. The algorithm integrates the improved S transform and Hough transform based on the aggregation criterion of the standard deviation of the sliding window, and completes the signal detection combined with the threshold based on the Neyman Pearson criterion. The simulation results show that the algorithm has good detection results below 14 dB. Compared with other algorithms, it is concluded that the algorithm has good detection ability under low SNR condition, does not rely on prior information, and has strong applicability.